fix: pin multiprocessing pools to the fork start method (Python 3.14) - #1439
Merged
Conversation
Python 3.14 changed the default start method on Linux from "fork" to "forkserver", which corrupts the model instances dynesty pool workers receive (factor-graph fits fail with "'Gaussian' object is not iterable") and silently degrades other pool creation to single-CPU fallbacks. A new autofit.non_linear.parallel.fork_context helper reproduces the pre-3.14 default on every platform (fork on POSIX except macOS, platform default on macOS/Windows) and is applied at every pool/process creation site: the dynesty pool (via a subclass, upstream hardcodes the default context), make_pool, SneakyPool/SneakierPool, the Process/Queue layer, the parallel EP optimiser, and nautilus (passed a pool object instead of an int so it no longer builds its own default-context pool). The dynesty single-CPU fallback log now includes the exception that caused it instead of unconditionally blaming the operating system. Fixes #1437 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the Python 3.14 blocker (#1437). Python 3.14 changed the default multiprocessing start method on Linux from
forktoforkserver; under forkserver, dynesty pool workers receive corrupted model instances (factor-graph fits fail withTypeError: 'Gaussian' object is not iterable) and other pool creation silently degrades to single-CPU fallbacks. Workspace scripts run at module level without__main__guards, so fork-inheritance semantics are load-bearing across the whole workflow.A new
autofit.non_linear.parallel.fork_contexthelper reproduces the pre-3.14 default on every platform —forkon POSIX except macOS; the platform default (spawn) on macOS (its default since Python 3.8; fork-with-threads can abort there) and Windows (no fork). It is applied at every pool/process creation site:_fork_pool_clssubclass — upstreamdynesty.pool.Pool.__enter__hardcodes the default context)AbstractSearch.make_poolSneakyPool/SneakierPooland theProcess/Queuelayer (covers emcee/zeus)pool=<int>, so it no longer builds internal default-context pools (also gives their cleanup a deterministicwithblock)The dynesty single-CPU fallback log now reports the exception that caused it instead of unconditionally claiming the OS doesn't support multiprocessing.
API Changes
None — internal changes only.
fork_contextis a new helper inautofit.non_linear.parallel; no existing public symbol, signature, or default changes. Runtime behaviour on Python ≤3.13 is identical (the pinned context equals the old platform default everywhere).See full details below.
Test Plan
test_autofit/non_linear/test_fork_context.py(numpy-only): fork pinning of the helper andProcessclass, plus a factor-graph likelihood evaluated in a real fork-pool worker asserted equal to the in-process value (the shape-collapse guard for fix: pin fork start method for pools (Python 3.14 forkserver default) #1437).test_autofitsuite in the worktree: 1641 passed, 2 skipped.PYAUTO_TEST_MODE=1 PYAUTO_SMALL_DATASETS=1 python autofit_workspace/scripts/overview/overview_1_the_basics.py— previously failed at the factor-graph fit, now completes end-to-end.Follow-ups after merge (separate PRs, per #1437): re-add 3.14 to PyAutoHands
python_matrix.yml, add 3.14 classifiers to the libraries, retarget the PyAutoNerves 3.14 experimental banner.Full API Changes (for automation & release notes)
Added
autofit.non_linear.parallel.fork_context()— returns the multiprocessing context PyAutoFit uses for all pools/processes:forkon POSIX except macOS, platform default elsewhere. Reproduces the pre-Python-3.14 default on every platform.Changed Behaviour
make_pool) use theforkstart method instead of the newforkserverdefault — restoring Python ≤3.13 behaviour. No change on Python ≤3.13, macOS, or Windows.Generated by the PyAutoLabs agent workflow.
🤖 Generated with Claude Code